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ABS-TRACT ' x , 

Spj^cific evaluation design. is a non trivial problem 
which must be developed through a se| of, procedures involving 
considerable study of the enterprise to be evaluated. Three criteria 
to be considered when an eValuatoir provides data for a decision maker 
are:. (1) the perc,entage of the evaluation data that are actually used 
by~^ th^ decision mak^r; (2) the percentage oJf the decision maker's 
decisions that are made^^-in part^ with the^use of the> evaluation 
data; and (3) the degree to which the dat#is utilized in* the most 
iTmportant decisions^ ^v^n where all decisJLons are not made using the 
evalua-feion data. These (Sjriteria have been named efficiency^ 
completeness^ and focus^ .respectively. Evaluators should avoid ^ 
errors^ ind prefer operational to non operational techniques. 
Svaluator/decision maker interaction patterns should be incorporated , 
to the evaluation* design.^ (Author/BJG) 
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Introduction , . ' , * 

v^^^^^^^^^^tn-^^ecea^^ there has been considerable activity in the develop- 
ment of mS'thodology foKevaluation (Scrivettj (1967), Stuff lebeam et al. 
(197L), Provus (1971), Benedict ^^3a)) . Most of these new syst^ 
propose a process for -the development of an evaluation design which ^s^^ 
specific to the project, program, curriculum,/ or" enterprise to be evaluated 
The pirotffess of designing the evaluation is viewed as part of the evaluation 
activity rather than something that preceeis thk beginning of evaluation. 

■ Although there are important differences among these new approaches, 
most are in agreement that the specific evaluation design is a non trivial 
problem which .must be developed throfigh a set of . procedures involving 
considerable study of the enterprise to be evaluated.' Eyaluators using 
such ^pproacR^s find-it difficult to respond to^requests for proposals 
which ■■ require the proposal to -specify the specific evaluration design and, 
sometimes, even the measiiting instruments ,to. be used. In, order to respond 
they either have to do one fourth or more of the total Work for nothing 
and with no guarantee. of being accepted for the remainder of the iwork ot\^ 
abandon their -professional belief s and propose -somfe pre-designed plan . 
•like a pre-post experimental group—quasi control group design, using 
some specified standardized Instruments as the measures to be employed. 
Many of these 'evaluators see the- use of pre-deslgned techniques, either 
In the design or In Instrumentation, as inappropriate to the purpose of 
evaluation. 

Inappropriateness^f Pre-Designed Techniques 

•A large number of evaluators have as .the purpose for their work: to 
' .provide data for deci'tfiop making (Stuff lebeam (1969)). -"Some of these ; 
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evaluators define this purpose in terms of -three criteria: the percentage 
of t"he evaluation data that are actually used by the decision maker to 
*fhom-they are provided;' the' percentage of ""k decion make's "decisions that 
are made, in part, with the use of 'evaluation data;- and .where not all 
decisions are made using evaluation data the degree to which the data are 
provided for the more important decisions (Hutchinson (1972)). 

. These criteria have been named, efficiency, compl'eteness , and focus 
.respectively.- Coffing, et al. .(1973) have pitovided operational procedures^ 
for measuring these criteria. Completeness' is the percentage of the 
decisions of a decision maker that are made, in part, with the use of 
evaluation 'data. Each, decision maker is asl^ed to keep ^ log recording ^ 
each decision that he or sh6 makes and what data, if ^^y, were used in 
\iaking the decision. The evaluator keeps a log of the evaluation data 
' that are provided to each decision maker. At* regular intervals the 
evaluatqr by .inspection of the log of decisions can determine whether or 
not each decision was madd, in part, with" the use of. evaluation data. 
The percent of the' decisions that were made using evaluation data is We 
percentage of completeness. , • - ► < 

The percentage of completeness that can be obtained is limited by 
the amount of resource? available to provide the decision maker with data. 
. These evaluators are trying to maximize the -percentage of completeness 
within the resource constraints. Less than this represents error of 
completeness. . • " 

•Where the percentage of completeness is less than,100%, focus is the 

degree to which data were prqvided Sor the more important decisions. At^ 

■ ' ' . / . 

regular intervals the evalifator^ can ask the decision maker to prioritize 

/> 

the. decisions on the log by the criteria of importance. The evaluator 
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can construct a two bv two contingency table with categories of , evaluation 
data used, evaluation data Tlot uged and most important, least important. 
The marginal for used, not used is the^ marginal imposed for most important, 
leist important. A percentage of correct f<)cus may then be calculated. 
These evaluators are trying to design" an ev.aluation that will be 100% 
foQused on the most important decisions. Less than this represents error 
6f focus. / 

Efficiency is the percentage o£ the efyaluation data which are used 
in making decisions. At regular intervals , the evaluator by inspection 
of the log of decisions can determine whether or not each evaluation data 
^ report, variable by variably has been used by the decision m^ker in 
making^any deaisions*. The percent of the datf provided that were actually" 

i . ' - ' ' 

used is the percentage of efficiency. THese evaluators are trying to , 
design an evaluation that will have 100% efficiency. Less than this 
represents* error of efficiency. . • ' '» . ) ' , > 

. The use of pre-designed techniques Is potentially .a considerable • : 
' source, o^ error of efficiency. In the. case of the quasi control group 
design the decision maker may reject the usefulness of the data on- the 
b\sis that* the control group is either known to be (or simply might be) 
diffejrent in some systematic way from the experimental group. ^ the 
' ^ case of the use of a pre-desigited measuring instrument; for exampl^, a > 
^ standardized self concept, te^t; the decision maker m^y find the data not 
to be useful because upon inspection of the 1: est the decision maker f inds ^ 

that it does not have face validity with respect to his or her meaning of 

* * 

self coneept. ^ It is only when the specific evaluation design, is worked^- 
out with the decision, maker that the evaluator can be sure that these 

threats to the^- efficiency of the ^evaluatipn design can* be avoided. 

• **/*' * • * « 

» ' 

^ -.'^ ,' . . ' ■ ' ' < . , 

■ , ■■ ^ i • » • 



The po'tential sojirces of erroV of efficiency just mentioned can be 
categorized as bias Was 'a lack of decision maker validity. Further, in_ 
these' situations th* decision maker was aware o^f the lack of decision 
maker validity. It is possible under some' circumstances, for there to fee 
a lack of deci-teion maker validity and for the decisign malcer to be- unaware^ 
of this bias. In this event 'the data will'lbe used when they shouldn't^be 
used. * 

Non Op.eratibnai Techniques /* , 

This kind of error is possible when the measuring instrument is not ^ 
operational. Imagine tha^ an evaluator. is' attempting to provide a project 
director with data for decision making. The project director has agreed 
to having the evaluator do direct observation in the classrooms using a 
rating form* One itep on the rating form is as follows: 

* The students air^'working in teaps — none, some, 
ai^ut half> most, all. ^ ^ 

, If the project director and ^the .evaluator observed t?he same classroom 

^t the same time they might not agree on ^^he observation. The evaluator 

migh^ count /the number of students working alone. The project director ; 

might counLjz^ie number of students in' groups o"f threa or more. Th^'Hwo • 

> 

observers have different ideas about what constitutes a .team. The 
evaluator would report more team activity than is accurate from tfie 
perspective of tl\e decision maker. *The project director might decide 
that everything is as he wants^it when^eve^ry thing is not as he Vants ipr. 
An ^operational item'would not have this problem. Consider the ' 

'following items t ; " * * ' ^ 

- V Count the number of students in groups of thre% (?r more. 

* Count the number of students in the-' room. 
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These items are not subject to misinterpretation, or misunderstanding. 

s ^The^ unbperational item is also subject to certain other types of 

* error of measurement » Consider again the item: . ^ . : 

The students are working in teatas — none, some, 
about half, most,^ all^ ^ 

One observer, may believe that school work , is quiet and not consider nois^ 

groups to be " working in teams." A different observer may believe that 

if it is quiet then the students are not really "working in teams" but 

rather' individually in a group. Therefore, this observer would not 'count 

qui^t groups. Non operational items permit observer reactivity. 

' Another possibility is that the observer begins with the first 

classroom assuming that noisy groups aren't really working. By the end 

■ of the day the noise has really gotten to the observer^ and now only 

totally silent groups get counted. 'Due to thi^ observer maturation next 

all classrooms have had the same standard applied- " ' " • 

'consider -a second iteA on the sara^ rating form. -.It is as follows: • 

The students are cooperating with each other — 

none, some,- about half, most, all. ^ * ^ \\ 

l< Observer who'beliexes thar school work should be quiet also believes \ , 

that the quiet students are cooperating with each other. Therefore, . 

this ot>server couiits the quiet g^'oups and quiet isolates. Since both 

items are. judged by a' ''level of noise" Standard the results are that'T^ 

two measures are not totally independent.' The other observer who likes 

t 

vigotous student interaction scores both items the same because cooperating 
is interaction in a group. For. this observer the it^ms are totaily 
dependent. The project director 'is^ getting one piece of information al- 
though he thinks he Is getting two separate pieces of inforinaticm. , 
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The use of operational items 'avoids these, k'inds of problems. The 

evaluator has the decision maker 6perationally define the intents or 

* 

goals of -the program. Tethniques can then be developed to measure the 
operational items. , • * » ^ 

Procedures for Avoiding Errors ' • ' ' - 

Procedures have been 'developed for providing data for decision making 
in such a way that the aboVe kin^s ofc^errors are avoided. ' The Fortune/ 
Hutchinson Evaluation Method6logy (Benedict (197-3), (1974)) provides for 
'regular interaction between the decision maker and .the evaluator in the 
design of the ei^aluation; The procedures retire the approval of the 
decision maker at t*^ end of each phase of design. 

This methodology also provides for operationally defining the decision 
maker's intents (Hutchinson and Benedict (1970)) so that operational 

measurements can be designed. These procedures are designed i^ an attempt 

'• T " , ■ - ' 

to maximize the criteria for ^kiciency, completeness, and'focus. They . 

also provide a process for evaluating the evaluation' design itself dn 
terms' of these criteria. "^^v ' 

The F/E Evaluation Methodology has been subjected. to methodological 
research by Jones (1971)-, Gordon (1973), Benedict (1973b) ,• Rosen (1974), . 
' and Jeffers (1974). In ea,c}i 'case "the intpnt .of the research was to identify 
problems with the procedures.^ In each dase probleiiis were found and solutions 
have been proposed or are under' development . Unfortunately, a proposed ^' 
-symposium to present the results of this methodological research on evalu- 
'ation vas turned down by -Division D of AERA. - ^ ' 
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